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ASSTRAi 

^ There exists the practical possibility of rdndoaily 

selected control groups for oiitdoae assessments in mental health 
programs. Ethical considerations for randomization include 
distinguishing bettween innovation and Jfoolluq around,^ and 
discriminatory implications of a first-come first-served selection 
procedure^ Practical considerations include reviewing the differences 
between volunteers ^and hon- volunteers and the utility c'f rigorous 
evaluation ' information. The strategies for itplenenting control 
groups in the research -design, using administration and staff, » 
^ncli/de: (1) use of minimal program group; (2) separation of acfion 
and evaluation units; (3) inclusion of the lottery procedure on the ' 
informed consen^form; and (4) allowance for particularly needy 
applicant^. For/aealing with participants the strategies include: use 
of a^o-^tery fort, selection; informing potential participants about 
lottery; contacting and informing participants personally; when 
possible, providing subsequent services Jbo the no-program group. 
(BN) ^ ' . ^ , ' . 
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Some Answers for Evaluative R«8«archera- 



Jaclc'^IlcKllllp and / Jacqueline K. VoBB 
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Our presentation focuses on the putcome assessment of social service 

programs, whether educational, theri^peutlc or growth*orlent<|^« Such assess-* 

oents ask If participants were better off after the program than they 

would have been l£ no or so^^dlf ferettt program had been offered. The 

assumption tinderlylng our j^resentatlon Is that this question cannot be 

« answered Absolutely, Evaluative data^ like all social science Information, 

rely 6n (a^ least Implicit) comparisons In order to hay^^meanlng. Whether 

or tiot a^ program's 25X cure rate is Impressive or disappointing depends on 

cure rates for similar programs. Especially In times of ln<;reased compe-) 

tition for social^ welfare dollars,' statementjP^uch as '*lf only one is helped" 

cannot be taken seriously. 

Accepjtlng the implicit comparative nature of evaluative data, the best 
•> » ' 

comparison for Judging the postgrogr am status of participants is provided 

by the status of individuals as\slmllar as .possible to programs participants 

i e^iceptlng tliat they h^ive not participated in the program (although they 

may have participated in some alternative program) . The question cannot 

be put simply: ^'Did the program do good?" but rather "Did the program do. 

better th^M|||||y>'> oi 'How did the progr/ n co^i- *:o some other program?" 



th^H^mjL^'' oi 'How did the progr/ n co^- 
'^^1^^^ groups are those randcHft' 



^« *"**^^B^^^ >roup8 are those randcHft' m the saipe population 

aa tnostf randomly selected for the progfram. 



While moet evaluative researchers would accept this presentation as 
far as It goes, many remain unconvinced of the practical jibsslblllty or 
necessity of randomly selected comparison groups for outcome assessments 
(e.g., Osttlnc, 1976; or Crdnbach, 1977), For exampla, one survey^f fed^^^ 
•rally funded outcome evaluations of social service prof rams (Bemstaln fc 
Freeman/ 1975) found that only one third utilized randomly selected co^mrl* 
ton groups, while 43% did not Include any comparison group. Drawing on 
oiir .xpcrlence In Inplementlng a nunber of evaluations (McKllllp, 1973; . 
rlcKllllp & Kanens, 1977; Vo88» 1976), It la our purpose to present InforM- 
tlon which will Increase the practical possibility of the use of rigorous 
evaluation deelens. Recognizing that the choice of design In evaluatlinft 
research Is not dictated soley by scientific concerns but, as In all other 
aspects of program evaluation, Is strongly influenced by political factors, 
we review both ethical and practical arguments whlqh we have found helpful 
In gaining acceptance of rleorous desifena and then provide soae "how to" 
, suggest iotas. 

- * y 

Ethical Arguments ' i 

An extremely useful source in this area iff Robert Boruch's (1976) 
collection of counter argjiaents to common objections to randomization. We 
have been quite successful in overcoming ehtlcal objections by taking an 
aggreejiive rather than a defensive posture using the following arguiMnta. 

When new programs or program alterations are Introduced, human service 
organliatlons need to distinguish between innovation and fooling around 
(Cl^rt, Light & Hosteller, 1975). IThUe change is a positive organisa- 
tional characterlstl^it must be planned in ways which allow assessment 
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of itn Talu<,^tt.gv» unbiased comparisou to othar possible changas i^/or 
to tha aatabllsbad program. The nedlcal literature provides a nuolier of axap«>li 
of Ixmovatlon which are Initially attested to by glowing clinical teporta, 
but later upon more rigorous evaluation, are found to be useless or. even - 
Imiaiful (Gilbert, et al, , ^75). "Innovation need not be slowed but needs 
to be 4otte In a responsible manner. (A more whimsical approach can only 
leed to charges of quackery and to attempts at outside ragu^^atlon). if the 
fttlth of program ^mlnlstratoirs and staff in the value of the change la 
not absolute and If plausible arguments for negative (aide) effects can be 
devlaed^ the thrust of thl^ approach can be quite forceful. 

► Iprpngolng programs, a point of discussion can be the (usual) first com 
*l'ft-««rv« (J^PS) entry process. Especi^illy where the potential deaand 
•'^••ds the number of openings, we have been successful in arguing that the 
FCFS ••lection procedure shows a bias toward those In the know, with reaily 
«cc«^^ to normal coipmunlcation channels to the most assertive, «^ile thoa^ 
•liflblebut less well connected or newer to the established system tend to be t 
•keluded. It is 8peci|^.cally this typ". of criticism that was, raised most 
recently rhoxst services provided by community mental health centers bjr Nader's' . 
R^«rs <Chu & Trotter, 1974). When wide advertisement coiri)ined with outreach 
•ttorts result^ In an excess of applicatlonTfor the program affair and specifi- 
able selection procedure is random selection of program partidpatns from all 
•llglbles with those not selected servlnp, as a no-prograft comparison iirour 
practical Arguments 

When part-lcipation in programs is voluntary, the ^||nteer wj.ll difi.r 
'•y»tfcatlcaU» from the non-volunteer. People do not buy Into a program 
rando^y but only on the basis of a particular need attraction. Such 
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: p«opl« ar* not static beings And thus will probably mature or ijwrova to som 

I ■ 

■ extant even without a progrn. An example of this process Is providad by 
Voss's (1978) evaluation of a sex-education workstibp vhlch included a randomly 
chosen no-program comparison group equal In size to the nuid)er of applications 
the program. received above the enrollment celling. While program participants 
showed significant Increases in knowledge over a. one month period, so did ^ 
the no-progi;am group' ! in this case the bias caused by using a comparison 
group of non-partlc^ant, non-volunteers would have resulted in the Illusion 
of a program effect when none existed. A similar bias, \but in the opposite 
direction, results when program participants are deteriorating rather than 
iaproving (in the absence of a progiram), e.g., drug abuse or alcohol treat- 
ment programs. In this case the uJe of a comparison group from* different 
population could easily cover-up a /real program effect. 

A sbmevhat similar point is tkat volunteers usually differ from non- * 
volunteers in levels of performanJe, so that for both groups, post-progrM statue 
needa ta be compared to pre-progrim statui x;ln order to gauge p«<^ram effects. 
For example,HcKilllp and Kamens 1(1977) found that volunteers for a health 
education progrem had lower blooJ pressure, ate less Junk food and engaged 
in more exercise than non-voluntiers. In this case, mere comparison of poet- 



pr-ogram status for the two eroupk woul 



lead to the Inference of a program 



effect where no such inference js JuBtjlfied. (See also Garrard, 1972; and 
Zuckermen, Tushup, & Tinner, 1970.) 

The st^ngast practical arLument tor rigorous evaluations rest^ on 
the^tlllty of the resulting li/fonnation. As the ii^ortance of\he decision 

'^^^ *" *^ '° likelihood of program 



8ucce.8. or where viable alternative approaches ejdst, rigorous designs' are 
womt useful. There Is little argument with the assertion that the best. 
. the clearest , the, most persuasive information on program effects come from 
the use of Experimental designs. Where this type of information is needed v 
rigorous evaluations are mos-t practical.. 

A numbct of related points can be made. As Rossi (1972) points out, 
in our .••modlfjSd" welfare state, with a high level of both mobility and 
literacy, strong hurilan service effects are hard to find. Farther, the more 
difficult it is to show an eftiact, the more necessary it is to use a 
rigoroiw ' design. In evauatton of the Sftlk .pollo vacdn* a quasi-experimental 
. design, comparing treats^ volunteJrs with en Undifferentiated noh-tteated 

group untlerestlaated thL effectiveness of the vaccine by. 50Z compared to an 
. eiqperlnental design using a randomly selected* non-treated group (Ileler, 1972). ' 
Secondly, if we do not follow the conq)arl8on group as closely as we do the - 
Program group. It may happen that the comparlsonlroup will have received a 
"proxy" program without our knowl«Ige. 'This miy ie a problem for evaluatl as 
of all designs, but will probably be less serlouawhere the compafison group 
is specified and measured both before and ^fter the program.' Pinalii, It 
has been a frequent experience in evaluatlo^ that comparison groups which 
are not chosen randomly Introduce a negative JsIm, i.e.. their use '^s it 
■ore difficult to show a prpgran effect. Both because'^of the tendency to 
select comparison groups of superior ability and because of a related tenden^\ 
for measures to be more reliable for superior groups, a number of prominent ' 
evalMations.have concluded th^t program effects did not exist fwhen such effect, 
probably did, ^e.g.. Head Start and perforntonce contracting. 
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HOW TO DO IT 



Da^llnp vith Ad j„ini,trator« and Staff . ^ : 

Sioce a^^avorabl* ethical and practice atmowhara for a rlgorou. : 
•valuation exi«t« when ther|^are nor6 Eligible appllcanta than .the^progra- 
can handle. utUliatlon Infoirmatlon la very laportant. While adnlnlaeratorA- 

■ajr be receptiv, to increased advertising of the prograaw racep'tlvlty wi^^ \ 
be aided if uiiderserved populations can be IdeniElfled. Thla ^formjtlM^ 
wlli serve the addltlbi4l function of dl'rtctlng adv^rt^.lag a^oatreach" ' . 
ef forta. ; ' ' ' - > 

While 4 oo-program comparison group has many advantages. It Is. not 
alway. the only ,or best comparison group. An attractive «,ditioB or alternative 
le a minlaal program grcup. I.e.. one receiving ^he low-coat purely mfor- 
Mtlonal componecvts of the full program. The use of thla group «li<„« the 
etaff to feel 'they are reaching more people, rather than withholding a 
needed pro^tam firbm them. but. at the sa^e tli^e. Is'not'a (athlcally dubious) ' 
pl«^«bo group ^hlch deceives participants by creating the Wresslon of 
helping byt net actually providing assistance. Ceabirfed with the no-progra. 
group, the use of a minimal i*rogral group allows adminl|tratora to gauge ' 
the over*!! effect of the full program as weiraa tlie ben««it^ Its 
expensive coinponents. McKllllp (1978) discusses th^ advantages iid. dis- 
advantages of vB^latlona on the minimal program. A final point . Is that 
the uae o*-^i, no-Program group' .&kea clear to the |,taf f ihat It is the progra. 
and not the staff which Is 'being evaluated. ' 

. Three other strategies facilitated Interactions with aaWnlitrators and ' 
ataff. First, .ction 'and evaluation units were kept s,,parate, but malnt^ilned 
cloae contact. S^cpndlyV inclusion of description of th.'lpttery (randoW.«tio„) 
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for ••lActlon of progiraBi participants was included on th^ signed infotned conatot 

fonaa. Thia procedures eased. administrative feara of rlitrlbutlon from thoae 

ualgned to norprogram groupa. And thirdly, a nutaber df pr pgr aa slots 

f ' ' '* ' 

^•re kept open for thoae applicants 'whom the ataff felt were particularly ' 

- » 'I * • • » f . 

needy or deserving* Patacfrom these participants waa exdtided from analyaea - 
of outcome measures. ^ 

Pealing with Program Particip antg . ^ 

^ Perhaps because^^Aw^ within a university settitig we have encountered 

. little'. resistance to the use of randomization from potential program partlcipanta 

' (although considerab^le^resiatance came from program administrators and ataff). 

We^found the foljlowlng procedures useful, however. 
■ J — * 

^ First, the term randomization is never used. Rather/ program partl^cipaivta 

ere selected by lotterjr^wUflln there are more applicants than available poai^lona^ 

We have found the concept of a lottery t,o be .acceptable and widely underatood. 

Perhaps this effect would be limited to states whi<*h have lotterlea.. 

^ \ ^ ^ • . 

Secondly, we have informed potential participants at the tfae of application 

ebout the lottery and about the minimal or no-program groups. Including 
thla Information on the informed consent form has minimized reseatment and 
the pctenitial political hassle caused by randomization out of an attractive 
program (Wortman, Hendricks, & Hellis, 1976) • 

Thirdp aa much as possible, .we have dealt with appllcanta 'personally, 
aapacially thos.€ijas signed to no-p^pgram groups. Those refused eltry were 
informed by a phone call which l^icluded discussion of the rationale for the V 
lottery. This procedure allows us to answer questions and clear up mlaunder- 
^/-^tandlnga ai sogn as pdsslble. .Fourth, when possible, the pwmlae of subsequent 
•ervlces to the no-program g^rouji' may^make program staff and group partlc^.pants 

■ i * • ' ; . ■ ' ■ ■ 

more comfortable. • 
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By using tills fotir pronW attack. I.e. lottery, fully infor»e<I^coii^«ie, 
psrsonj^ontact, And promise of .siibse^uent services. v« have been able to keep 
attrition, itom the no-program g^ups at thei same laval aa that' for t!ha 
program groupa. 
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